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Abstract 

In the K-12 education sector, learning objects are seen as important in 
providing quality resources for teachers and learners but there has been little 
formal research on the assessment of learning objects based on the qualities 
that would be important for K-12 teachers. In this paper we describe the 
developments in the K-12 sector, the arguments around learning object 
characteristics and the development of an assessment profile. We applied this 
instrument in two separate analyses of learning objects and found it useful in 
identifying characteristics of importance to teachers. 


Although there is an extensive and ever-growing literature about learning 
objects, the clarity of the term continues to be elusive (McGreal, 2004). The 
various approaches to learning objects attempt to meet two common 
objectives: (1) to reduce the overall costs of digital resources and (2) to obtain 
better learning resources (Wiley, 2003a). Downes (2001) contends: “the 
economics are relentless. It makes no sense to spend millions of dollars 
producing multiple versions of similar learning objects when single versions of 
the same objects could be shared at a much lower cost per institution” (If 4). 
Equally strongly, it can be argued that the provision of learning objects 
provides better access to quality resources and supports enhanced learning 
outcomes. Duval, Hodgkins, Rehat, and Robson (2003) explain the purpose of 
learning objects as “to increase the effectiveness of learning by making 
content more readily available, by reducing the cost and effort to produce 
quality content, and by allowing content to be more easily shared” (If 1 ). 

These two purposes, effectiveness and efficiency, receive differing emphases 
from different sectors. 

Wiley (2000) provides an overview of the development of the term learning 
object and comments that “the proliferation of definitions for the term ‘learning 
object’ makes communication confusing and difficult” (p. 5). He illustrates his 



contention with a number of examples of different terms developed by various 
repositories for learning objects, such as, “pedagogical documents” (used by 
ARIADNE), “educational software components” (ESCOT), and “online learning 
materials” (MERLOT). Similarly, in reviewing papers fora ED-MEDIA pre- 
conference workshop, Duval and his colleagues (2003) concluded that “quite 
a few papers devoted considerable space to a discussion of the exact 
definition or nature of learning objects as a concept” and that “many groups 
seem to be grappling with issues that relate to the pedagogically sound use of 
learning objects” (p. 2). Two years later, the pedagogical issues unresolved 
but pedagogical criteria now identified by IEEE, the literature has moved on to 
focus on databases and repositories. 

Differing perspectives have been taken to the concern with producing learning 
resources that can be used more than once. In general, some definitions have 
focused on the “object” part of the term while others have emphasized the 
“learning” aspect. Learning objects do not have value or utility outside 
instructional contexts. Their value is in their application to classroom settings 
and to online environments where teachers may or may not be present. As a 
result, learning objects are designed to help teachers perform these functions: 

• introduce new topics and skills 

• provide reinforcement to existing skills 

• extend learning by providing new means for presenting 
curricular material 

• illustrate concepts that are less easily explained through 
traditional teaching methods 

• support new types of learning opportunities not available in a 
classroom environment 

• provide enrichment activities for gifted and highly motivated 
students. 

Mortimer (2002) contends that while we may be surprised that there is no 
single definition of learning objects and that the term holds different meanings 
for different people, most learning objects have four characteristics. First, 
each learning object has a learning objective, content and activities that 
support the objective and assessment activities that reflect that expectation. 
Second, they usually take less than 15 minutes to complete. Third, the content 
is metatagged to some set of standards, and finally, the object can exist on its 
own and be provided to the learner in a just-in-time and as-needed fashion. 
However, there is still disagreement about these components. Merrill (2002) 
insists that objects without a learning design component are knowledge 
objects only. Until recently metatags did not include any instructional design 
components and McGee (2003a) points out that “most educational metadata 
do not include attributes for evaluation, suggesting that documenting of 
learning occurs outside the learning object experience” 4). In addition, she 
commented that “there are still many unanswered questions, particularly in the 
area of pedagogical design” flj 1). 

Most of the pedagogical questions have focused in two areas: first, whether a 
specific learning design should be used in designing the object and second, 
the issue of reusability. The pedagogical issue was raised as more designers 



adopted a constructivist approach to learning. Some developers were 
concerned that too much specificity increased the size of the object and 
thereby reduced its reusability. Some wanted to include resources but not 
provide a learning design within the object. Others felt the previous learning 
models were adequate. None of these views includes the desires of an end 
user. The most recent position has been one approaching pedagogical 
neutrality. McCormick (2003) reflecting his experiences with CELEBRATE 
argues that efforts to include a specific pedagogy are doomed to failure and 
advocates for “the development of LOs with sophisticated, high quality media 
representations of content, around which teachers build learning activities and 
assessment” 2). Reusability has been of major interest over this past year, 
much of it focused on metatagging fatigue and the development of 
alternatives and the issues surrounding databases and repositories. While this 
is of particular concern in other sectors, in the K-12 sector the development of 
high-quality approved curriculum materials that meet the learning needs of 
students is generally welcomed by teachers so that reusability is less of a 
concern. 

Learning Objects in the K-12 Sector 

The overall development of learning objects is still much of a ragged front. 
Much of this development in Australia, Canada, Europe and the US, has 
occurred in the post-secondary sector, mainly in universities and much of the 
research focuses on issues in an adult setting. There are a number of 
examples of learning object repositories or research-based working groups 
that are trying to research, design, collect and disseminate learning objects. 
The important point to note is that it is important that there be sufficient use 
and reuse of objects in practice to help inform the pedagogical theories and 
technical standards which have been identified. The development of learning 
objects is a complex undertaking and a number of institutions who formed 
consortia to help share development have revealed that after the first flush of 
enthusiasm it has been difficult to sustain interest without having a central 
guiding body to ensure sufficient practical application (EOE History, 2003, ^ 
10 ). 

The context for employing learning objects differs between the K-12 sector 
and postsecondary sector and affects the design of learning objects. Unlike 
more adult settings where learning objects can be used by learners 
independently, much of the current efforts in the K-12 sector have been to 
design learning objects for face-to-face classroom settings. This change in 
context and the requirement to design for incorporation into pre-existing 
learning strategies has created additional complexities for designers who must 
take into account the culture and climate of school classrooms. 

In Australia and New Zealand, Canada, the USA, the UK and Europe, there 
have also been developments in the K-12 sector. For the USA curriculum, 
there are over 18 general databases of what could loosely be identified as 
learning objects, most with minimal meta-tagging. These might be considered 
more like online repositories that contain artifacts, websites, and lesson notes, 
as well as learning objects. They range from sites with sample learning 



problems to Education World and from SMETE (Science, Mathematics, 
Engineering and Technology Educational content and services) to the Apple 
Learning Exchange. This latter repository contains a range of lesson plans, 
lesson starter ideas, student projects, virtual field trips, videos and interviews. 

In Canada the initiative for the development of K-12 learning objects has been 
led by the work of the learnalberta.ca portal. It is now being used as the basis 
for development of a national portal under the Committee of Ministers of 
Education, Canada (CMEC). LearnAlberta is being developed to serve not 
only the K-12 but also the post-secondary and lifelong learning sectors. Much 
of the development of K-12 objects has been in the areas of mathematics and 
science and the total output is still relatively small. In 2003, Ontario 
announced its development of a similar portal that would be connected to the 
national initiative. In a review of learning object repositories for CANARIE, a 
national, Canadian government-funded, research committee, Porter, Curry, 
Muirhead and Galan (2002) noted that the projects were not yet at a sufficient 
stage of maturity to continue to advance without central leadership. They 
confirmed that an integrated focus was essential to prevent fragmentation in 
what is an expensive development initiative. Some of their analyses are 
appropriate outside the specifics of their project. For example, they noted the 
need to develop a critical mass of users and assessors and proposed the 
development of communities of practice to realise these educational goals. 
The notion of communities of practice can be traced back through the work of 
the MERLOT consortium in the US which uses peer-reviews of learning 
objects as the basis for inclusion. Additionally, they noted the importance of 
training and skill-development for teachers and the development of end-user 
applications as a further development in the use and reuse of learning objects. 

In the UK, in January, 2003, the government announced a web portal, 
CurriculumOnline, which is designed “to give teachers easy online access to a 
wide range of digital learning materials, which they can use to support their 
teaching across the curriculum” (Ifl )■ At this time the development of the 
portal is still in the early stages. A European initiative, the European 
SchoolNet ( www.eun.org ), is a partnership of over 26 European Ministries of 
Education interested in the educational use of ICT (information and 
communications technology) for policy-makers and education professionals. 
Based on the Canadian SchoolNet ( www.schoolnet.ca ), it coordinates 
discussions and activities among teachers, students, policy makers and 
commercial vendors. One of its projects was CELEBRATE (2002-2004). 

A large scale project, CELEBRATE was designed to examine how learning 
objects can enhance teaching and learning in European schools. It was 
funded by the European Commission’s Information Society Technology 
program (1ST) and included ministry personnel, university researchers, large 
educational publishers, content developers and technology suppliers. “Its key 
aim is to provide a large-scale test-bed or practical demonstration of how 
schools from across Europe can use, adapt, reuse and develop Learning 
Objects” (1j 5 FAQ). In the demonstration phase (January 2003-May 2004) 
teachers accessed reusable learning objects, and also the tools and a virtual 
learning environment to make their own objects, build courses, and 


communicate and collaborate with other developers. Over 1400 learning 
objects and 25 simple authoring templates were developed by commercial 
publishers and Ministries of Education. They were mainly in Mathematics, 
Science and Language with smaller numbers in other subjects. While 
assessing the demonstration project as generally successful, the evaluation 
team (McCormick, Scrimshaw, Li, & Clifford, 2004) concluded that the 
program would be sustainable, “only if at each stage of its development: 

• sufficient teachers want to use LOs 

• sufficient LOs are produced 

• the ones produced are the ones that are wanted 

• and the teachers are able to find and successfully use the LOs they 
want.” (p. 157) 

In Australia and New Zealand, under the initiative of The Le@rning Federation 
the development of learning objects while still in its early phrases is more 
extensive. There has been development of objects in a variety of areas, and 
initial feedback from teachers about their viability. In addition, the work of the 
Federation in exploring international exchanges of learning objects through 
“The Oklahoma Exchange” (2002) has helped document the difficulties faced 
in such an exchange. In particular, the work identified the importance of 
learning objects reflecting the Australian teachers’ pedagogical approaches to 
learning and the limited utility of objects which come from a transmission 
learning model rather that from one which has embraced the new learning 
principles. This makes the Australian project all the more important since this 
is a finding that is particular to the K-12 sector and unresearched elsewhere. 

Approaches to Learning Object Evaluation 

The evaluation of learning objects is a comparatively new concern as the 
quantity of learning objects has grown and the development of learning object 
repositories has come about to allow for greater ease in finding and using 
objects for both classroom and online instruction. The growth in the number of 
learning objects, the multiplicity of authors, their increasing diversity of design 
and their availability to trained and untrained educators has generated interest 
in how to evaluate them and which criteria to use to make judgments about 
their quality and usefulness. 

The need to evaluate learning objects requires the development of criteria to 
be used in judging them. Vargo, Nesbit, Belfer and Archambault (2003) 
developed a Learning Object Review Instrument or LORI to evaluate learning 
objects. The LORI approach uses the following 10 criteria when examining 
learning objects: 

• Presentation: Aesthetics 

• Presentation: Design for learning 

• Accuracy of content 

• Support for learning goals 

• Motivation 

• Interaction: Usability 

• Interaction: Feedback and adaptation 



• Reusability 

• Metadata and interoperability compliance 

• Accessibility 

The criteria were drawn from a review of pertinent literature on instructional 
design, computer science, multimedia development and educational 
psychology. Each measure was weighted equally and was rated on a four 
point scale from “weak” to “moderate” to “strong” to “perfect”. The LORI 
process involved both individual and group rating of learning objects. 
Reviewers are drawn from the discipline in which the object is intended for 
use. This “peer review process” closely mirrors other practices found in the 
postsecondary sector. It is unclear if this is or should be a requirement in the 
K-12 area. 

The criteria used by Merlot ( www.merlot.org ) to review learning objects for 
acceptance in its learning object repository include many of the same criteria 
used by Vargo et al.(2003). The Merlot process employs both individual 
evaluation (peer review) and referral to standards for learning objects. The 
standards or guidelines are an attempt to help reviewers assess materials 
submitted by faculty. The criteria used by Merlot reviewers fall into three broad 
areas: 

• Quality of Content including consideration of the quality of the 
specific information in the object and how well the content models 
the skills of the discipline 

• Potential Effectiveness as a teaching-learning tool : including the 
“actual effectiveness” of the object through personal use or making 
judgments about the potential effectiveness for improving instruction 
and learning by faculty and students 

• Ease of use: including consideration of the general layout of the 
object, the computer interface, attention to the buttons, menus, text 
and types of user-object navigation 

Peer reviewers use a five star scale. The scale describes a continuum from 
one star denoting “material not worthy of use” to a five star rating representing 
“excellence all around”. Like the LORI process, reviewers are drawn from the 
discipline for which the material is meant to be used. 

More recently, the Collaborative Learning Object Exchange (CLOE) based at 
the University of Waterloo in Ontario, Canada, has developed a peer review 
process for material developed through a collaborative initiative among the 
university and college sectors in Ontario for inclusion in a provincial learning 
object repository. This review process closely follows the MERLOT criteria 
but differs in the range and number of questions used in the review process. 
The Merlot criteria use a set of more than 30 individual questions requiring 
detailed answers, while the CLOE criteria use a smaller set of 
questions/criteria. The CLOE process also places the review process under 
the responsibility of an Editor-in-Chief who is responsible for managing the 
peer review process involving two categories of reviewers — instructional 
designers and subject matter experts. Reviewers in both processes are asked 
to evaluate the learning objects on the merits of quality of the content, its 


effectiveness as a teaching tool and its ease of use. The 14 items which 
comprise the CLOE criteria are listed below: 


• The content of the learning object is accurate. 

• The use of technology is appropriate for this content. 

• The content is presented clearly and professionally 
(spelling/grammar, etcetera). 

• Appropriate academic references are provided. 

• Credits to creators are provided. 

• There are clear learning objectives. 

• The learning object meets the stated learning objectives. 

• The target learners are clearly identified (academic level 
addressed/technical ability/demographics). 

• There are clear instructions for using the learning object. 

• The technology helps learners to engage effectively with the 
concept/skill/idea. 

• The learning object provides an opportunity for learners to obtain 
feedback within or outside the learning object. 

• The author provides evidence that the learning object enhances 
student learning. 

• Pre-requisite knowledge/skills, if needed, are identified. 

• The learning object stands alone and could be used in other learning 
environments. 

• The learning object is easy to use (i.e. navigation, user control). 

• The author indicates whether the learning object is accessible for 
learners with diverse needs. 

• Technical requirements for the learning object are provided. (Draft 
Review Guidelines 2003) 

Two significant reasons why the number of questions was reduced were to 
expedite the peer review process and to reduce the criteria to those deemed 
most critical to the successful use of the learning object. 

Different criteria have been developed for evaluation of software and interface 
design. Nielson (2003) developed ten criteria for examining user interface 
design and computer-user interaction. These can be used when evaluating 
learning objects. In the “Ten Usability Heuristics” of interface design at 
www.usit.com/papers/heuristic list.html Neilson closely parallels the Merlot 
and CLOE criteria but adds items that are important to learning in a 
technology enhanced learning environment. They include the following five 
criteria that directly bear on learning object design. 

• Visibility of system status: make certain the user always knows 
where they are in the learning object 

• Match between the learner and the real world: ensure the learner 
understands the object which uses words, phrases and concepts 
that speak the “user’s language”. 

• User control and freedom: ensure that if users choose a function by 
mistake, they can safely leave the unwanted state and that 



maximum flexibility can be offered regarding navigation through the 
object. 

• Recognition rather than recall: make sure that the user does not 
have to remember instructions for operating the learning object from 
one section of the object to another. 

• Help and documentation: make certain that necessary assistance is 
provided through the object and that such information should focus 
upon the user’s tasks and concrete steps to be carried out by the 
learner. 

The standards/criteria developed by MERLOT, CLOE, Vargo et al. (2003) and 
Neilson (2003) were developed for the postsecondary sector. There has been 
less attention paid to criteria for learning objects used in the K-12 sector. This 
lack of published material is best explained by the relatively few school level 
projects in the area of learning object development across the globe. The 
relative newness of projects such as the Le@rning Federation (AU), 
Curriculum Online (UK), CELEBRATE (EU) and LearnAlberta.ca (CA) has so 
far not attracted the attention of researchers. The lack of research has 
resulted in the development of project-specific criteria for review of learning 
objects. Another result has been the use of the criteria from the 
postsecondary sector and a growing concern regarding what should be added 
or subtracted for evaluating K-12 learning objects. 

A number of critical points for K-12 educators do not appear in lists of 
postsecondary criteria. For example, special requirements associated with 
child development and learning context do not appear. Learning objects used 
in a school or with school-age children require that attention be paid to 
concerns such as learner motivation and how the learning object supports 
independent learning and minimizes instructor intervention. In addition, other 
criteria not common in lists of postsecondary criteria include the need to 
clearly delineate what prior learning is required for use of specific learning 
objects, criteria regarding the concern over cultural bias, and the requirement 
that learning objects incorporate values that are consistent with prevailing 
community standards while also ensuring that the content is age appropriate. 

Postsecondary criteria also do not address the necessity for learning objects 
to include pedagogical information about implementation, information 
regarding multiple instructional settings, or information about using learning 
objects within existing instructional contexts. While help files are useful and 
necessary, guidelines about how to use learning objects are more beneficial 
for teachers. At this stage of adoption of information and communication 
technology, teachers require support and guidance concerning how best to 
use learning objects and computer technology in their teaching practice. 

Development of an Evaluation Instrument 

The criteria for evaluating learning objects were drawn from four sources: A) 
the CLOE draft guidelines, B) the Le@rning Federation Soundness 
Specification, C) the rating scale previously used by Vargo et al. (2003) in 
their Learning Object Review Instrument (LORI) and D) criteria developed with 



respect to the special concerns of the K-12 environment. The resulting 
Learning Object Evaluation Instrument (LOEI) was developed to examine 
school level content. The scale used in evaluating each component and of the 
subsequent total for each object IS NOT meant to provide a comparison 
among learning objects. The comparison of learning objects is fraught with 
complications when not all objects conform to similar designs or intended use. 
Rather, this scale allows a reviewer to determine the integrity, usability, 
learning, design and values focus of each object in and of itself. 

The 14 criteria used in this instrument and the principles they reflect are listed 
below: 


Integrity 

• The content of the learning object is accurate and reflects the 
ways in which knowledge is conceptualized within the domain. 

Usability 

• Clear instructions for using the learning object are provided. 

• The learning object is easy to use (i.e., navigation, user control, 
visibility of system status). 

Learning 

• Learning objectives are made explicit to learners and teachers. 

• The target learners are clearly identified (academic 
level/technical ability/demographics) and addressed. 

• Pre-requisite knowledge/skills are clear with connections to prior 
and future learning. 

Design 

• The technology helps learners to engage effectively with the 
concept/skill/ideas. 

• The learning object structures information content in order to 
scaffold student learning. 

• The learning object provides an opportunity for learners to obtain 
feedback either within or outside the learning object. 

• The learning object stands alone and reflects an awareness of 
the varying educational environments in which learning 
sequences and objects may be used by the learner. 

Values 

• The learning object is appropriate for community and cultural 
affiliations, including language, dialect, reading and writing. 

• Help and documentation files are provided for students and 
teachers including contextual assistance. 

• The design of visual and auditory information enhances learning 
and mental processes. 

• The learning object is accessible to learners with diverse needs. 

• The learning object does not require instructor intervention to be 
used effectively in a mixture of learning environments and 
learning sequences. 


Application of the Instrument 



Our analyses to assess the quality of the instrument and determine its utility in 
providing information for K-12 teachers occurred at two different times over a 
two year period. We worked with The Le@rning Federation’s learning object 
repository because it contained the most complete set of objects covering the 
greatest variety of subjects available designed for the K-12 sector. First, we 
used the instrument to assess 22 learning objects drawn from The Le@rning 
Federation’s databank of learning objects. Based on the outcome of that 
analysis we made minor refinements and then used it to assess a further 14 
learning objects one year later. 

We received web access to each object and its metadata for a limited period 
of time and independently used the criteria to do the assessments. We began 
by reviewing two objects. This allowed us to assess the clarity of the criteria 
and check the ease of use of the instrument. When we compared our 
analyses we agreed that the LOEI was easy to use and found a high degree 
of congruence in our assessments. We then used the criteria to review each 
object independently and did not assess one learning object against another. 

Overview of Findings from the Analyses 

Attempts to review learning objects are fraught with complexities not found in 
assessing other non-digital educational content. Learning objects are 
multifaceted, incorporating curricular content developed to meet state and 
national curriculum frameworks. Developers and designers face the need to 
design objects that are small (disaggregated) yet adaptable in design to allow 
them to be combined or recombined with other objects to create learning 
sequences (repurposing). However the disaggregated nature of learning 
objects requires that “content” be as flexible as possible in its design to 
encourage teachers to utilize objects across a variety of instructional settings. 
Hence, the task faced by learning object developers is to design learning 
materials that can stand alone (disaggregated) and be used in a variety of 
learning environments (repurposed). Therefore, an evaluation process must 
be sensitive to the overall goals that designers and developers have for such 
digital assets as well as the constraints upon designs. 

Learning objects differ from more traditional learning materials in a number of 
important ways. First, learning objects use a variety of media sources 
including text, graphics, sound, video and music. Unlike textbooks, learning 
objects are not developed for use in a universal format. While technical 
specifications for learning objects include minimum standards for operating 
systems, browser settings and necessary Internet browser “plug-ins”, the 
variety of hardware and infrastructure found across schools and the diversity 
within schools regarding classroom computers (e.g., large or small screens, 
high or low fidelity speakers)and differing levels of connectivity (i.e., 
bandwidth) add to the challenges that designers have in creating materials 
that can be universally accessed and used. This issue is compounded when 
educators expect to “repurpose” and “personalize” learning objects for 
individual teaching styles. In addition, the desire to “localize” learning objects 
for specific communities of learners only adds to the diversity and intricacy of 
design and technical considerations faced by designers. While technical 
standards address the notion of a common platform, with recommended 



settings and common infrastructure, experience suggests that the computer 
equipment used can have an effect upon the experience of the end user. With 
this knowledge as a basis this review we undertook to examine all the learning 
objects in a variety of Internet Browsers (Netscape 7.1 , Opera 7.1 1 and 
Internet Explorer 6.0). 

Learning objects also differ from more traditional educational content in that if 
the educational object is disaggregated sufficiently it will more likely be 
reused. This tension and/or desirability between large and small learning 
objects, self contained learning objects and medium sized learning object 
“chunks” is still less than straight forward for optimum utility. This tension 
between large and small objects is best explained in Figure 1.0, Instructional 
Context and Learning Object Design, which illustrates this tension and the 
requirement that learning objects, more often than not, require a context or 
“instructional wrap” to ensure that they enhance learning and can be used by 
instructors within a learning sequence. 


Figure 1. Context and learning object design 
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Concern for small flexible objects presents educators and developers with 
difficult decisions regarding how much content and learner supports should be 
incorporated into a learning object. This is an issue with many objects for the 
K-Year 12 environment where students may lack academic skills and/or 
experience to fully interact with learning objects in a purposeful manner. The 
challenge is “not too large” or “not too small” but “just right” to paraphrase 
Goldilocks in the well known nursery rhyme. In short, the size of object has 
enormous influence on how best to use learning objects. The learning objects 
we reviewed showcase current thinking in the area of size, complexity and 
flexibility. While some were quite large and self sufficient and existed as 
independent learning activities, other objects were best used within a 
contextual framework where teachers provided the instructional support for 
their use. Rather than see this as a criticism, the diversity of learning objects 
should be viewed as more analogous to a research and development 
strategy, where new understandings are created through the process of 
design, build, use, revise, use and document. 

Integrity 

The variety of learning objects provided by The Le@rning Federation 
highlighted the many kinds or types of “architectures” of learning objects that 
are emerging to address the specific needs of school-aged children. The 
approach to presenting content and the instructional strategies imbedded in 
the learning objects are congruent with practices found in Australian 
classrooms and incorporate best practices found in the literature on teaching 
in the various curriculum areas. For example, in some learning objects the 
design of the object, and the tasks presented to students to complete inside 
the object, reinforce the scientific method. Learners are asked to review the 
task, make an appropriate selection, make a prediction about the likelihood of 
success, test their prediction, compare their prediction with that observed in 
the simulated experimental setting and test their choice in a simulated 
situation. If unsuccessful, they are again led back to step one and if 
successful, on to the next task. The learners were led through an 
experimental design while the learner makes decisions and observes the 
effect that particular variables have upon the phenomena being observed. 
Such learning objects exemplify the criteria embodied in the “Integrity” 
standard. 

A variety of architectures concerning functionality and the treatment of content 
existed within disparate curricular areas. Some of the learning objects 
incorporated significant third party content resulting in a “narrative” design. 

The result was a much larger multimedia intensive learning object. In others, 
the learning design was best described as “explore and practice” where 
external third party content was not used and little multimedia audio or video 
(except animation) was employed. The two approaches highlighted both 
diversity of design approaches but also the use of specific architectures or 
models for specific curricular areas. This use of different learning approaches 
with specific curricula demonstrates how the developers have begun to build 
new knowledge about approaches to the development of leading edge 
learning objects. The variety of learning activities within learning objects 



included actions such as repurposing and augmenting story lines, creating 
personal/individual reports, recording experimental results and writing songs 
or poems. 

The use of game-like designs shows how gaming environments can be used 
for educational purposes. Educational gaming involves use of the narrative 
story line, complex, user-controlled navigation, the design of immersive 
environments, and inclusion of success indicators. As computer games 
continue to grow in popularity it is likely that learners will expect learning 
environments to include such activities. Early research suggests that boys in 
particular can benefit from such gaming environments (Foreman, 2004). 

Some learning object designs incorporated interdisciplinary experiences 
across curricular areas. In each, the pedagogical design and the subject 
matter content ensured they could be repurposed for use in a variety of 
curricular areas. This approach to designing learning objects for 
interdisciplinary reuse illustrates the potential of learning object design where 
in addition to small and generic designs which enable reusability 
interdisciplinary design enables reuse and repurposing across classroom 
settings, disciplines and age levels. 

Size or granularity is often considered to be in inverse relationship to 
complexity in the design of learning objects. However, the development of 
learning objects which are targeted for senior students suggests that the more 
advanced the audience and content, the larger some learning objects may 
become. Within the academic literature most authors suggest that learning 
objects should be small and focused on smaller portions of domain specific 
knowledge. It is thought that small learning objects encourage reusability and 
repurposing by teachers. However, learning objects for senior students which 
contain more advanced subject matter while also retaining the desired 
pedagogical features for K-12 learning objects, are likely to become more 
complex and may grow in size and density. 

As larger objects generally include more content components and 
predetermined pedagogies there is an increasing risk that this may reduce the 
instructional approaches employed by teachers. Consequently it is a matter of 
balancing competing approaches between larger more comprehensive 
learning objects which can border on units or modules and smaller learning 
objects that may not address all aspects of the knowledge or skills desired but 
result in learning objects that are more easily repurposed and reused. As 
additional features are added to learning objects, such as Help Buttons, 
demonstrations, and greater use of multimedia, the overall mass of learning 
objects may also grow. Without comprehensive field testing to gather data 
about the optimum size of learning objects, answers will remain illusive. 

Usability 

All online learning materials require clear instructions for using the learning 
object. In many of the learning objects, this was attended to through a 
combination of available help files or in “explanatory” objects that 
accompanied the learning objects. However, the question of whether it is best 



to include this material into the object thereby making it larger or to continue to 
disaggregate the files into different learning objects remains unresolved. 

All objects involve a level of familiarity with computers and with input devises 
such as keyboards, mice, tab key functions and space bars. Certainly, it will 
be inevitable all students will one day have familiarity with computer 
technology. But today that is not true. Therefore, it is highly recommended 
that a learning object be developed to focus on how to use and interact with 
learning objects. This learning object would not only support accessibility, but 
also usability and navigation. 

Clear learner navigation through the object is essential for student 
independence in working with learning objects. One important area is to 
ensure that a system status icon is included in all learning objects. In most 
cases the first step in using the learning object requires the learner to click on 
a button or link to begin loading the applet. Not all objects include a system 
status button to allow the user to see the progress of loading. This may cause 
confusion regarding feedback about the status of the connection or the 
operation of the object. 

Attention to navigation design is equally important. A repertoire of functional 
capabilities should include the use of “Help” buttons throughout the object, 
“Back and Forward” buttons, and the use of “demonstrations” to instruct 
students about how to interact with and make use of the learning object. The 
use of “Loading Buttons” in most learning objects to inform users how long 
and how fast the object was loading from its source (repository, local server 
etc.) demonstrates the growing understanding surrounding “interoperability” 
and the disparate connectivity found within many jurisdictions. Feedback on 
successful progress through the learning objects that included the use of both 
audio clues and visual clues to signal to the learner appropriate or 
inappropriate movements and/or decisions was also important. 

Learning 

All learning objects incorporated specific learning objectives in their design. 
However, the objectives were not always as explicit to learners or to teachers 
as they could be or perhaps should be. The lack of detailed curricular 
information regarding what particular learning outcome the object was meant 
to address requires attention as to how to embed this information or metadata 
into the object. Without this vital information, teachers will be required to make 
instructional decisions regarding where these objects should be used within 
an instructional setting without benefit of curriculum maps and the onus would 
be upon teachers to deduce the learning outcomes for which the objects were 
designed. 

Students also need clear learning objectives regarding the purpose of the 
objects. In many of the objects this was presented but some provide learners 
with directions concerning how to use the object but they did not specifically 
state what the students would “learn” or what outcomes they would achieve 
when finished with the learning object. 



One criterion of the LOEI was the necessity to clearly identify academic level, 
necessary technical abilities, and specific demographics for learners to 
interact with the learning object. This information should appear not only in the 
online repository but also in the learning object itself. This recommendation for 
additional information to be available about the learning object could be 
addressed through the addition of a teacher help file in the object. 

Alternatively, the information could be included in the online site through the 
development of a teacher user file that could be downloaded and made 
available to teachers. 

It is essential to provide teachers with information regarding what prior 
knowledge learners must have to successfully complete the tasks presented 
in the learning object. The design of one object required the learner to have 
prior knowledge about a topic and the object did not expose or present 
information about it prior to asking learners to do a task based on that 
information. However, once learners successfully complete the tasks they 
were exposed to enrichment information. As a result, learners must have 
significant prior knowledge to use this object. However, nothing in the 
information found in the repository notes, or in the introduction to the object or 
in the object control files provide information about this. It is conceivable that 
this omission is purposeful and that this object is meant to be used to test 
prior knowledge or as an assessment learning object. Nevertheless, without 
some instructions the object could conceivably be used inappropriately by 
learners or teachers. 

Design 

One of the chief benefits attributed to the development and use of learning 
objects is the ability of multimedia technology to present students with 
learning opportunities not easily replicated in classroom environments. The 
inclusion of the criterion item, “The technology helps learners to engage 
effectively with the concept/skill/ideas,” was incorporated into the LOEI 
instrument to provide some assessment about the use of technology to 
support expanded learning opportunities. Objects provided learners with 
opportunities to manipulate variables not as easily presented in the 
classroom. Moreover, the objects also allowed students to return to the object 
many times to review the skills presented or to reinforce new learning. Some 
learning objects also demonstrated the potential that sharing “object design 
shells” can have in supporting the laddering of skills from the simple to the 
more advanced. 

Learning objects present opportunities to extend and expand learning 
activities to beyond the confines of the object itself. Therefore, one of the 
design considerations and evaluation criteria used in LOEI is to assess to 
what extent the learning object structures information content in order to 
scaffold student learning. Objects could use help or hint files, hypertext or 
pop-up instructions to provide just in time assistance. Scaffolding that 
encouraged self-assessment and reflection is also encouraged. Many objects 
included scaffolding that provided additional information or next steps hints. In 
this regard, we found that the objects reviewed did not provide links to 
external resources. This lack of reference to outside resources was no doubt 



intentional, allowing the objects to be “repurposed” by a variety of teachers 
and used in a variety of curricular areas and settings without referral to “other” 
more particular and less general resources. Not referring to external 
resources in the learning object alleviates the task of constantly reviewing the 
external links to ensure that Internet sites do not change their addresses or 
post inappropriate content. One solution to these twin problems is to set up 
mirror sites for external materials/resources thereby maintaining secure 
Internet addresses while also ensuring that content does not change without 
their knowledge. 

The addition of an external reference site could be used to build a shared 
resource site/repository of online materials from educators. This site could 
include tips concerning implementation, lesson plans built around the specific 
learning objects, a site for sharing student work, and a location where groups 
of educators could develop and share “peer reviewed” set of links for use by 
students and teachers when using objects to build learning sequences or 
activities. This community resource would assist learners and teachers to 
provide enrichment activities. 

The inclusion of multiple challenge levels is a desirable design feature which 
serves the dual purpose of providing additional activities while also scaffolding 
learning within a single object. It also serves the purpose of further reinforcing 
learning experienced in the initial set of learning activities. Still used relatively 
infrequently, this option should become a new functional instructional design 
feature that may be incorporated in other learning objects 

Learners require feedback mechanisms to ensure that they can self assess 
whether they have acquired new skills, knowledge or attitudes presented by 
the learning object. Not all objects contained assessment opportunities. In 
some, tasks are supplied and feedback regarding correct answers is 
immediately presented. Hints are provided. Some learning objects help 
learners review where they have succeeded or where they have made 
mistakes. One design feature not all objects shared allowed students to print 
their answers and engage in off-line activities through the construction of a 
jigsaw puzzle or students could print their results and refer to the data at a 
later date. Print functions facilitate students in capturing data generated within 
the object and provide the opportunity to record and capture project materials 
or to store and reflect on the materials in the learning object. The challenge for 
designers will be to expand this capability to link online with offline activities. 

A proposal which arises from the postsecondary and corporate training 
sectors, is that learning objects be designed to be used without direct 
instructor involvement in either face-to-face or online learning environments. 
For this reason one criterion for evaluating learning objects is their ability to 
stand alone i.e., be a “chunk” that does not require direct teacher 
administration thereby enabling students to work independently but with some 
ongoing interaction between instructor-learner-object. Many of the learning 
objects had this balance. They could be used by students in a classroom 
setting independent of teacher administration, yet they relied upon teachers to 
weave these objects into existing teaching practices. In some objects where 



there is an insufficient introduction to the learning objectives more can be 
done to meet this criterion. 

Values 

The appropriateness of language, image and sound is a key criterion in 
learning objects for the K-12 sector. Educators expect that the materials will 
be free of racial or gender stereotyping and that the images reflect the variety 
of groups in the society. Care must be taken to include appropriate language 
use and to reflect the community’s social norms. 

It cannot be over-emphasized how important comprehensive help and 
documentation files are to good design of learning objects. Learners should 
not be confronted with inadequate assistance regarding how to use a learning 
object. Help should also be contextual and provide specific assistance for the 
task being undertaken. This often requires effort and therefore greater cost, 
yet it is critical to “good design” principles. Help files need to have sufficient 
information about how to move through the learning object. 

Good design includes an engaging interface. The use of figures to personalize 
the objects for students is often included. The use of text-to-voice features to 
reinforce the text on the screen is another desirable feature. It reinforces 
instructions and expands the ways in which students can interact with the 
learning objects. This also contributes to greater accessibility for learners with 
diverse needs. 

Learners with diverse needs require special attention. In some cases it may 
be a physical handicap while in other instances the challenge may be 
intellectual. Designs that use alternative input strategies such as keyboards 
rather than relying exclusively on a mouse, or navigating across screens using 
the tab button, or hearing rather than reading instructions support diverse 
students. So too does the use of animated graphics to help students navigate 
through learning objects or assist learners to use objects without relying upon 
highly developed reading skills. The increased use of audio instructions and of 
animated instructions can only increase the accessibility of these objects to 
“students at risk”. This use of audio and visual clues is important for both low 
literacy students who may find text based feedback difficult to comprehend or 
for visually impaired students who find text-based directions difficult to see. 

The final item in the evaluation instrument concerns the extent to which 
learning objects are “appropriate for community and cultural affiliations, 
including language, dialect, reading and writing”. This can be the most difficult 
and potentially the most problematical of the criterion to quantify. Yet to ignore 
or to skip over this criterion is inappropriate. Consequently, it was included in 
the LOEI instrument. In evaluating learning objects using this criterion, the 
reviewers asks the question, did the learning objects portray a particular 
group, or individual, or use stereotypical representations of individuals or 
groups or include graphical designs or voice-overs that could cause concern 
in the widest variety of learning settings or among the cultural groups in the 
country of origin of the learning object? Within the K-12 sector where learning 
objects are seeking approval for curriculum use, the appropriate use of 



language and cultural norms and the importance of inclusion and 
representation of diversity are values that must underlie the content and 
processes involved. Other criteria include minority language rights, recognition 
of indigenous peoples’ rights and environmental agreements. 

Discussion 

From our analysis of the objects, three topics stand out. These are (1 ) the 
accessibility criterion; (2) the student interface and (3) pedagogical issues. 

Accessibility 

From discussions with developers we became more cognizant of the 
complexity of practical design issues involved in addressing accessibility 
criteria. Incorporating all accessibility features into all learning objects made 
the learning objects more complex to develop, more costly to create and in 
limited cases adversely affected the overall design of the learning object. An 
alternative was to develop separate purpose-built learning objects for learners 
with specific special needs. Everyone subscribes to the principle of universal 
design but unresolved practical issues remain: Should all adaptations be 
included in every object or are some adaptations for specific needs best 
addressed through purpose-built objects? Will some students choose the 
adaptations to avoid working through the object directly? These questions are 
best resolved through actual observations with teachers and learners. 

The Le@rning Federation had completed some preliminary assessments of 
learning objects with teachers (Snapshot 2, 2004) who were involved in an 
introductory workshop prior to trialling the learning objects in their classrooms. 
Teachers worked in pairs to evaluate 30 learning objects and then participated 
in a general discussion of their utility for their classrooms. One of the most 
common concerns for teachers is the level of language used. This was also 
the case in this instance. The large majority of learning objects were 
considered suitable and probably going to be used but some teachers were 
concerned that in some objects the language level was too high or that it was 
too text-based and required extensive reading. 

In a subsequent field review across 14 schools (Lake, Phillips, Lowe, 
Cummings, Schibeci, & Miller, 2004) teachers demonstrated how they 
adapted learning objects to suit the particular needs of individual children. 
Being able to loop back through the object to repeat sequences or view 
alternatives was viewed as a way to allow for different levels of difficulty which 
enhanced the object for the teacher. All K-12 learning objects have to be able 
to meet a range of learner competencies and learning styles. The provision of 
language level options and alternative pathways then were two design 
features which could receive particular attention from designers and should be 
included under this criterion. Commonsense suggests a balanced approach to 
the issue of accessibility. This leaves developers with the issue of addressing 
additional accessibility criteria on an object by object basis. 



Student interface 

As we interacted with the learning objects we tried to remain aware that we 
were imposing adult perceptions on materials that would be used mainly by 
children. Kenworthy (1993) has proposed eight guides to assist in high quality 
interface design. These involve keeping the cognitive load appropriate to the 
age and grade level of the learner, and avoiding dividing attention between 
text and graphics. Instead he suggests that audio rather than text should 
accompany images. He proposed multiple representations, color, icons and 
animations to direct attention and enhance communication, and concept 
maps, table of contents and other visual means for assisting knowledge 
retrieval. Practice exercises to encourage rehearsal, realistic simulations, 
exercises that reinforce and sustain interest and use of concrete words with 
audio support to assist with abstract terms are other points Kenworthy 
identified. 

We went back to the field trials data from Lake and his colleagues (2004) and 
found that there were some instances of students commenting on the 
importance of variety, especially when they reused the object a number of 
times. We also reviewed comments from teachers who found that rather than 
answering the question students put in any keyboard character in response to 
a prompt so that the program would continue to the next screen. Engagement, 
interactivity and immediacy were all identified as important to students. 
Equally, we remain convinced that other students have had little access with 
computers and need to be helped to develop a repertoire of skills surrounding 
the loading and use of learning objects. This may mean that for the first 
iteration of learning objects, additional help buttons and prompts are needed 
to assist these learners. 

Pedagogical issues 

One of the major advantages of learning objects is their interactivity. In 
reviewing the CELEBRATE project, McCormick and his colleagues (2004) 
noted that “there is an interplay between the affordances of a learning object’s 
pedagogy and the pedagogical practices the teacher is able to construct, 
given her pedagogic competence in using learning objects, and her underlying 
conceptions of learning” (p. 134). This range in professional competence 
based on familiarity with learning objects was also evident in the report from 
the Le@rning Federation’s field trial. Teachers unfamiliar with their use 
provided learning objects as stand-alone unconnected activities undertaken 
by all students at the same time or used them as a whiteboard activity in a 
teacher-directed full-class lesson. Teachers with greater experience with 
learning objects embedded them in a sequence of activities that encouraged 
student-oriented individual and group learning. 

McCormick’s (2004) argument is that teachers will use learning objects in 
alignment with their own pedagogical orientation and that using specific 
orientations may be a detriment to use of the object or the object will be 
placed in the larger context of the teacher’s own orientation. He contends that 
“learning objects can enhance active learning by students, collaboration, 
authentic material and activity, and provide multiple perspectives in 
knowledge” but goes on to point out that regardless of intention “It is clear that 



the locus of control or design of pedagogy is in the hands of the teacher” (p. 
134). 

At the same time much of the educational literature supports a change from a 
more teacher-directed to student-oriented inquiry model and the integration of 
technology is no longer seen as an optional skill for teachers or learners. 
Perhaps the provision of learning objects will help resolve this false 
dichotomy. As Parrish (2004) notes, 

To the extent that the learning object movement can foster effective 
learning by introducing active learning experiences, supporting student- 
centred learning environments, propagating new ideas about instruction 
and increasing collaboration and sharing of resources, it can play a 
major role in improving education and training, (p. 65) 

In conclusion, we recognize that this is a fruitful area for further inquiry. Each 
of the three dimensions discussed above require further research and learning 
objects in general will benefit from more field-based research involving 
students and teachers in the process. 
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